Google Revamps Crawler Documentation with New Structure and Enhanced Content Google has unveiled a major revamp of its Crawler documentation, streamlining the main overview page and dividing the content into three more focused pages. Despite the changelog downplaying the changes, a new section has been added and the entire crawler overview page has essentially been rewritten. The additional pages allow Google to enhance the information density of all the crawler pages and improve topical coverage. What’s New in the Documentation? While Google’s documentation changelog notes two changes, there is actually much more encompassed in this update. Here are some of the key changes: – Updated the user agent string for the GoogleProducer crawler. – Included content encoding information. – Introduced a new section about technical properties. The technical properties section includes completely new information that was not previously available. There are no changes to crawler behavior, but by creating three specialized pages, Google can add more information to the crawler overview page while simultaneously making it more concise. This update also provides new details about content encoding (compression): Google’s crawlers and fetchers support gzip, deflate, and Brotli (br). The supported content encodings for each Google user agent are listed in the Accept-Encoding header of each request they make. For example, Accept-Encoding: gzip, deflate, br. Additional information is supplied about crawling over HTTP/1.1 and HTTP/2, alongside a statement about the goal of crawling as many pages as possible without impacting server performance. Goal of the Revamp The documentation change was prompted by the sheer size of the overview page. Adding more crawler information would have made it even larger. Therefore, the decision was made to subdivide the page into three topics so specific crawler content could keep growing, leaving room for more general information on the overview page. Creating dedicated sub-topic pages is a smart solution for better serving users. This is how the changelog describes the change: The documentation became very extensive, limiting the ability to expand content about Google’s crawlers and user-triggered fetchers. The reorganization included adding specific notes about the impact each crawler has on products, and a robots.txt snippet for each crawler to demonstrate using the user agent tokens. No significant changes were made to the content otherwise. The changelog downplays the changes by referring to it as a reorganization, as the crawler overview is significantly rewritten, alongside the creation of three brand-new pages. Although the content is largely the same, its division into sub-topics allows Google to continue adding more content to the new pages without further expanding the original overview. Now a true overview, the original page “Overview of Google crawlers and fetchers (user agents)” has more granular content on standalone pages. Google has published three new pages: 1. Common Crawlers 2. Special-Case Crawlers 3. User-Triggered Fetchers 1. Common Crawlers Common crawlers related primarily to GoogleBot, like the Google-InspectionTool, which uses the GoogleBot user agent, are covered here. All bots listed adhere to robots.txt rules. Documented Google crawlers include: – Googlebot – Googlebot Image – Googlebot Video – Googlebot News – Google StoreBot – Google-InspectionTool – GoogleOther – GoogleOther-Image – GoogleOther-Video – Google-CloudVertexBot – Google-Extended 2. Special-Case Crawlers These crawlers are associated with specific products and agreements with those product users, operated from IPs separate from GoogleBot’s. The list of special-case crawlers includes: – AdSense (User Agent for Robots.txt: Mediapartners-Google) – AdsBot (User Agent for Robots.txt: AdsBot-Google) – AdsBot Mobile Web (User Agent for Robots.txt: AdsBot-Google-Mobile) – APIs-Google (User Agent for Robots.txt: APIs-Google) – Google-Safety (User Agent for Robots.txt: Google-Safety) 3. User-Triggered Fetchers This page describes bots activated by user requests, such as Google Site Verifier, which operates under user requests, like a site on Google Cloud retrieving an external RSS feed. Generally, these fetchers disregard robots.txt rules. Google’s crawler’s general technical properties apply here as well. Covered User-triggered Fetchers: – Feedfetcher – Google Publisher Center – Google Read Aloud – Google Site Verifier Takeaway: Google’s overly comprehensive crawler overview page may have become less useful due to its size. The more concise overview now serves as an entry point from where users can delve into more specific subtopics. This change illustrates how to refresh a page that might be underperforming due to its comprehensiveness. Breaking a comprehensive page into standalone topics addresses specific user needs and potentially increases their usefulness in search results. This update doesn’t reflect changes in Google’s algorithm but showcases an update for enhancing documentation usability and future-proofing for content expansion. Explore Google’s Newly Organized Documentation: – Overview of Google crawlers and fetchers (user agents) – List of Google’s common crawlers – List of Google’s special-case crawlers – List of Google user-triggered fetchers

SEO Send an email October 14, 2024

0 6 3 minutes read